Determination of relative position of an apparatus

ABSTRACT

This application describes fixture having a body and a plurality of fiducials with at least one of the fiducials having a first state and a second state and able to move to one of the first state and the second state. The fiducials are arranged such that knowledge of the 3D positions of the fiducials in an observing coordinate system is sufficient to determine a first position and a first orientation of the body in the observing coordinate system.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. application Ser. No.16/044,321, filed Jul. 24, 2018, which claims priority to U.S.application Ser. No. 15/483,626, filed Apr. 10, 2017, now U.S. Pat. No.10,078,908, which claims priority to U.S. Provisional Application No.62/374,277, filed Aug. 12, 2016, all of which are incorporated herein bythis reference.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not applicable.

BACKGROUND Field

The present invention generally relates to determination of relativepositions of equipment and fixtures in a workcell.

Description of the Related Art

Robotic workcells commonly use a robot arm to move items betweenlocations, for example between a transport tray and a processingfixture. In current systems, the locations from which the item is pickedup and where the item is to be placed must be known precisely by thecontroller of the robot arm as the robot moves blindly betweenpositions. Set-up of a workcell therefore requires either precisionplacement of the fixture relative to the robot arm in pre-determinedpositions or calibration of the system after the robot arm and fixtureare in place. In either case, the robot arm and the fixture must remainin their calibrated positions or the workcell will no longer functionproperly.

Methods of determining the position of an object in space bytriangulation are known.

Methods of using two cameras to take pictures of the same scene, findparts that match while shifting the two images with respect to eachother, identify the shifted amount, also known as the “disparity,” atwhich objects in the image best match, and use the disparity inconjunction with the optical design to calculate the distance from thecameras to the object.

Tracking of a moving object may be improved by use of an active fiducialsuch as disclosed in U.S. Pat. No. 8,082,064. The sensed position of thefiducial may be used as feedback in a servo control loop to improve theaccuracy of positioning a robot arm. Fiducials may be placed on therobot arm and a target object, wherein the sensed positions of thefiducials are used as feedback to continuously guide the arm towards thetarget object.

SUMMARY

It is desirable for a robotic workcell to autonomously determine thelocations and orientations of each item in the workcell and adjust theprogramming of the robot arm so it can successfully interact with eachof the items without operator intervention or a manual calibration ofthe positions of the modules.

It is further desirable to provide the ability to locate items usingpassive camera systems and avoid the need for illuminators, scanninglasers, or devices that emit structured light. This improves thesimplicity of system, avoid the cost of additional components, andeliminates risks associated with the use of lasers or other emittersaround people.

It is desirable to enable a workcell to use a robot module that does nothave a fixed or pre-defined location and orientation with respect to acamera module.

It is desirable during set-up to convert information about the positionof fiducials, modules, or interaction points that is known in acoordinate system associated with a camera module into a differentcoordinate system that is associated with a robot module. Once thisinformation is converted to the coordinate system of the robot module,control of the robot arm and interaction of the robot arm with theinteraction point can be accomplished without further input from thecamera module. In particular, the camera module does not providecontinuous information on the position of any object or module.

It is further desirable for a fixture to carry all the necessaryinformation for its use in an autonomous workcell in an embedded memorythat can be downloaded automatically by the workcell processor.

It is further desirable for a fixture to have a body and a plurality offiducials fixedly coupled to the body. At least one of the plurality offiducials comprises a first state and a second state and is configuredto selectively move to one of the first state and the second state. Thefiducials are arranged such that knowledge of the 3D positions of theplurality of fiducials in an observing coordinate system is sufficientto determine a first position and a first orientation of the body in theobserving coordinate system.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are included to provide furtherunderstanding and are incorporated in and constitute a part of thisspecification, illustrate disclosed embodiments and, together with thedescription, serve to explain the principles of the disclosedembodiments. In the drawings:

FIG. 1 depicts an exemplary work cell according to certain aspects ofthe present disclosure.

FIGS. 2A-2B depict plan views of the robot module from the work cell ofFIG. 1 according to certain aspects of the present disclosure.

FIG. 3 depicts a plan view of a portion of a workcell according tocertain aspects of the present disclosure.

FIGS. 4A-4B depict two exemplary optical designs according to certainaspects of the present disclosure.

FIG. 4C depicts a third exemplary optical design according to certainaspects of the present disclosure.

FIGS. 5A-5B depict an exemplary fixture and its associated spatialenvelope according to certain aspects of the present disclosure.

DETAILED DESCRIPTION

The following description discloses embodiments of a system and methodof identifying the location and orientation of a robot arm, tools,fixtures, and other devices in a work cell or other defined volume ofspace.

The detailed description set forth below is intended as a description ofvarious configurations of the subject technology and is not intended torepresent the only configurations in which the subject technology may bepracticed. The appended drawings are incorporated herein and constitutea part of the detailed description. The detailed description includesspecific details for the purpose of providing a thorough understandingof the subject technology. However, it will be apparent to those skilledin the art that the subject technology may be practiced without thesespecific details. In some instances, well-known structures andcomponents are shown in block diagram form in order to avoid obscuringthe concepts of the subject technology. Like components are labeled withidentical element numbers for ease of understanding.

As used within this disclosure, the term “light” means electromagneticenergy having a wavelength within the range of 10 nanometers to 1millimeter. In certain embodiments, this range is preferably 300-1100nanometers. In certain embodiments, this range is preferably 700-1100nanometers. In certain embodiments, this range is preferably 2-10micrometers.

As used within this disclosure, the term “fiducial” means a device orassembly that has two states that can be differentiated by observation.In certain embodiments, the fiducial changes color. In certainembodiments, the fiducial exposes a first surface while in a first stateand exposes a second surface while in a second state. In certainembodiments, the fiducial emits a first amount of light while in a firststate and emits a second amount of light that is different from thefirst amount of light while in a second state. In certain embodiments,the fiducial is a light emitter that is “on” in a first state and “off”in a second state. In certain embodiments, the fiducial emitselectromagnetic energy at wavelengths within the range of 10 nanometersto 1 millimeter. In certain embodiments, this range is 100 nanometers to100 micrometers. In certain embodiments, the fiducial emitselectromagnetic energy over a defined solid angle.

As used within this disclosure, the term “constant” means a value of aparameter that changes less than an amount that would affect the commonfunction or usage of an object or system associated with the parameter.

FIG. 1 depicts an exemplary workcell 10 according to certain aspects ofthe present disclosure. The workcell 10 comprises a camera module 100, arobot arm module 200, a fixture 300, and an electronics module 400. Incertain embodiments, a computer 500 or other remote access system iscoupled to the electronics module 400 to provide a user interface. Incertain embodiments, the workcell 10 may include a work surface or othersupport structures that have been omitted from FIG. 1 for clarity.

The camera module 100 has a field of view (FOV) 130 bounded by edges 132and having an included horizontal angle 134. The FOV 130 has a verticalaspect as well that is not shown in FIG. 1 for sake of simplicity. Thecamera module 100 has a coordinate system 160 with orthogonal axes C1,C2, and C3. The camera module has two optical systems 120, 122 withinenclosure 110. Each of the optical systems 120, 122 has a sensor (notvisible in FIG. 1) that is positioned such that images of items withinthe FOV are formed on the sensors. The position on the sensor of theimage of an item within the FOV of the optical system is referred toherein as the position of the item on the sensor. Information about theposition of an item comprises one of more of a two-dimensional (2D)location of a centroid of the image on the planar surface of the sensor,the maximum intensity of the image, the total energy of the image, thesize of the image in one or more of a width, height, or dimension of ashape such as a circle, or other comparative characteristic of images ofa target item, for example an illuminated fiducial, on the sensor. Acentroid may be determined an intensity-weighted location or the centerof an area determined by the intensity exceeding a threshold.

The information about the position of an item on the sensor of theoptical system 120 and the position of the same item on the sensor ofthe optical system 122 is stored as a position data pair in a memory,for example the memory of camera module 100. This information may bejust the 2D location of a feature of the image, for example thecentroid, or a multi-pixel color image, or other optical derivatives ofthe image formed on the sensor.

Each sensor has associated electronics that provide an output containinginformation about the position of a target item on the sensor. Eachoptical system 120, 122 has its own FOV wherein the FOV of opticalsystem 122 partially overlaps the FOV of optical system 120 and theoverlap of the individual FOVs of the two optical systems 120, 122 formthe FOV 130 of the camera module. In certain embodiments, the cameramodule 100 includes one or more of a processor, a memory, signal andpower handling circuits, and communication devices (not shown in FIG.1). In certain embodiments, the memory contains information about thearrangement of the optical systems 120, 122. This information comprisesone or more of the distance between the optical axes (not visible inFIG. 1) of the optical systems 120, 122, the angle between the opticalaxes of the optical systems 120, 122, the position and orientation ofthe sensors of the optical systems 120, 122 with respect to therespective optical axes and to the coordinate system 160.

The robot arm module 200 has a base 210 with a top surface 230, an arm220, and a fiducial set 240 comprising at least three fiducials 242,244, and 246 mounted on the top surface 230. The robot arm module 200has a coordinate system 260 with orthogonal axes R1, R2, and R3. Incertain embodiments, the robot arm module 200 comprises a processor 270and may additionally comprise a memory, signal and power handlingcircuits, and communication devices (not shown in FIG. 1) that arecoupled to the processor 270. The processor 270 is communicativelycoupled to the processor in camera module 100. In certain embodiments,the memory comprises information about the 3D positions of the fiducialset 240 within the coordinate system 260.

The fixture 300 has a base 310 having a top surface 330, a contactelement 320, and a fiducial set 340 comprising at least three fiducials342, 344, and 346 mounted on the top surface 330. The fixture 300 has acoordinate system 360 with orthogonal axes F1, F2, and F3. In certainembodiments, the fixture 300 comprises a processor 370 and mayadditionally comprise a memory, signal and power handling circuits, andcommunication devices (not shown in FIG. 1). The processor of thefixture 300 is communicatively coupled to the processor 370 of cameramodule 100. In certain embodiments, the memory comprises informationabout the 3D positions of the fiducial set 340 within the coordinatesystem 360.

A workcell will have a particular function or process to accomplish. Inthe example workcell 10 of FIG. 1, the robot arm module 200 willmanipulate an object (not shown in FIG. 1) and interact with the fixture300 to perform an operation on the object. This interaction can only beaccomplished if the relative position of the point of interaction on thefixture 300 is known to the processor 270 in the coordinate system 260,which would enable the processor 270 to manipulate the robot arm 22 toposition the object at the point of interaction on the fixture 300. Inconventional systems, this is accomplished by fixing the fixture 300 inposition relative to the robot arm module 200 and precisely determiningthe point of interaction in the coordinate system 260. This is a lengthyand time-consuming calibration process. If the position of either therobot arm module 200 or the fixture 300 are moved even slightly, thecalibration process must be repeated.

In an exemplary embodiment, an operator initiates a “determineconfiguration” operational mode of the workcell after the workcell isreconfigured to perform a new operation. The processor of theelectronics module 400 determines what modules are communicativelycoupled to it. The camera module 100 is activated to observe the workarea of the workcell 10. The processor of the electronics module 400retrieves information from the memories of the camera module 100, therobot module 200, and the fixture 300. Based on this information, theprocessor of the electronics module 400 selects a calibration method andmanipulates the camera module 100 and the fiducials of the robot module200 and the fixture 300 to determine the positions of at least a portionof the fiducials of robot module 200 and fixture 300 in the coordinatesystem 160.

The processor of the electronics module 400 then determines thepositions and orientation of the coordinate systems 260, 360 incoordinate system 160, thus providing a capability to transform, or“map,” any position defined in either of coordinate systems 260, 360into an equivalent position defined in coordinate system 260. Theinteraction points of the fixture 300, which are retrieved from thememory of fixture 300 and defined in coordinate system 360, are mappedinto coordinate system 260. The processor 270 of robot module 200 nowhas all the information that it needs to move the robot arm 200 to anyof the interaction points of fixture 300 without further input from thecamera module 100.

Once the configuration of the workcell 10 is known, the workcell can beoperated in an “operational mode” to process parts. In certainembodiments, the output of camera module 100 is now turned off and,thus, cannot provide continuous feedback on the absolute or relativepositions of the robot arm 200 nor fixture 300. In certain embodiments,the camera module 100 may remain active but workcell 10 is operatedwithout updating the 3D position and orientation of the first modulewith respect to the second module. Only when the position of one of themodules 200, 300 is detected to be changes, for example by sensingmovement using an accelerometer coupled to a module, is the informationabout the position of the fixture 300 in the coordinate system 260updated.

In certain embodiments, the camera module 100 comprises a binoculararrangement of the optical systems 120, 122 that enables thedetermination of the position of any single target item, for examplefiducial 342, in three-dimensional (3D) space within the FOV 130 andfocal range of the camera module 100. The workcell system 10 isconfigured to unambiguously identify the target item in each of theimages formed on the sensors of the optical systems 120, 122, as isdiscussed in greater detail with respect to FIG. 4A. Once the positionson the sensors of the optical systems 120, 122 of the target item areknown, the 3D position of the target item in the coordinate system 160can be determined through trigonometry and knowledge of the arrangementof the optical systems 120, 122. Information about the arrangement ofthe optical systems 120, 122 comprises one or more of the distancebetween the optical axes, a relative angle between the optical axes, thedistance between and relative angle of the sensors, the orientation ofthe optical axes relative to a mounting base of the enclosure 110, andthe position and orientation of one or more components of the cameramodule 100 relative to the center location and orientation of thecoordinate system 160.

For the fixture 300, once the positions of at least two of the fiducialsin the fiducial set 340 are known, the orientation of an axis passingthrough these two fiducials can be determined. This is not sufficient,however, to locate the fixture 300 or, more importantly, the point ofinteraction of the fixture 300 in coordinate system 160. Additionalknowledge of the construction of fixture 300, in particular thepositions of the fiducials 342, 344, 346 relative to the point ofinteraction, must be known.

In certain embodiments, the memory of fixture 300 contains informationabout the 3D positions of the fiducial set 340 defined within thecoordinate system 360. In certain embodiments, one of the fiducials offiducial set 340 is positioned at the center (0,0,0) of the coordinatesystem 360 and a second fiducial of fiducial set 340 is positioned alongone of the axes F1, F2, and F3. In certain embodiments, the center andorientation of the coordinate system are offset in position and anglefrom the fiducials of fiducial set 340 and the information about theoffsets and angles are contained in the memory of fixture 300.

An exemplary method of determining the position and orientation of thecoordinate system 360 within the coordinate system 160 starts bydetermining the positions in coordinate system 160 of at least twofiducials of fiducial set 340, for example fiducials 342, 344. Thisprovides a point location, for example defined by fiducial 342, and adirection vector, for example defined by the vector from fiducial 342 tofiducial 344. Combined with the information about the positions offiducials 342, 344 in coordinate system 360, the offset position of thecenter of coordinate system 360 from the center of coordinate system 160along the axes C1, C2, C3 and the coordinate transformation matrixdescribing the rotational offset of the coordinate system 360 around theaxes C1, C2, C3 can be calculated.

The information about the positions of fiducials 342, 344 in coordinatesystem 360 is essential to determining the position and orientation ofthe coordinate system 360 within the coordinate system 160. In thesystem 10, the electronics module 400 comprises a processor that iscoupled to the optical systems 120, 122 and receives information aboutthe positions of the images of the fiducials on the sensor of eachoptical system 120, 122. The camera module has a memory that containsinformation about the arrangement of the optical systems 120, 122. Thefixture 300 has a memory that contains information about the 3Dpositions of the fiducial set 340 defined within the coordinate system360. The processor of the electronics module 400 is communicativelycoupled to the memories of the camera module 100 and the fixture 300 andconfigured to be able to retrieve the information from both thememories. In certain embodiments, this retrieval is triggered when thesystem 10 is powered on. After information is retrieved from allconnected systems, e.g. the camera module 100, the robot module 200, andthe fixture 300 in the example system 10 of FIG. 1, the processor of theelectronics module 400 initiates a “position determination” process thatincludes the steps described above for determining the position andorientation, which can also be expressed as a coordinate transformmatrix, of the coordinate system 360 within the coordinate system 160.

Determining the 3D position of each fiducial in coordinate system 160requires the unambiguous determination of the image of the fiducial onthe sensor of each of the optical systems 120, 122. An exemplary meansof doing so is to configure the fiducial to emit light in a certainrange of wavelengths, for example infrared light, and select a sensorthat is sensitive to this same range of wavelengths. Filters can beprovided in the optical path of the optical systems 120, 122 to blocklight outside of this range. If a single fiducial is turned on to emitlight, which can be considered a “first state,” with all other fiducialsturned off, which can be considered a “second state,” then a single spotwill be illuminated on the sensors of optical systems 120,122.

In certain embodiments, the sensors of optical systems 120, 122 arePosition Sensitive Detectors (PSDs) that determine the position of alight spot in the two dimensions of the sensor surface. A PSD determinesthe position of the centroid of the light spot and its measurementaccuracy and resolution is independent of the spot shape and size. ThePSD may also provide information related to the maximum brightness ofthe light spot, the total energy of the bright spot, and one or moreaspects of the shape and size of the bright spot. Compared to an imagingdetector such as a Charge-Coupled Device (CCD) imager, a PSD has theadvantages of fast response, much lower dark current, and lower cost.

In some circumstances, the FOV of camera module 100 may includeextraneous sources of light, for example an incandescent light behindthe fixture, having a wavelength within the sensitive range of thesensors. An exemplary method of discriminating the fiducial from suchextraneous sources is to modulate the frequency of the light emitted bythe fiducial. As a PSD is a fast device, modulation frequencies in thehundreds of Hz are feasible and therefore avoid the 50/60 Hz frequencyof common light sources as well as unmodulated light emitted by sourcessuch as the sun and thermally hot objects. The output signal of the PSDcan be passed through a filter, for example a bandpass filter or ahi-pass filter, that will block light having frequencies outside of thesensitive range of the sensors.

Another exemplary method of determining the 3D position of multiplefiducials in coordinate system 160 is to selectively cause a portion ofthe fiducials to move to the first state and cause the rest of thefiducials to move to the second state. For example, at least threefiducials in a common fiducial set are turned on, for example fiducials342, 344, 346 of fiducial set 340, while turning off all other fiducialson that fiducial set as well as all other fiducial sets in the workcell10 and using an imaging sensor, for example a CCD imager, to capture a2D image of the FOV of each optical system 120, 122. The two images willrespectively have a plurality of first positions on the first sensor anda plurality of second positions on the second sensor. The images areprocessed to identify the 3D locations of each fiducial that is turnedon. If the relative positions of the fiducials 342, 344,346 are known,for example in a local coordinate system, a pattern-matching algorithmcan determine the orientation and position of the local coordinatesystem, relative to the observing coordinate system, that is required toproduce images of the three fiducials at the sensed locations on thesensors.

These determined locations of fiducials 342, 344,346 in coordinatesystem 160 form a 3D pattern that can be matched to a 3D pattern basedon the information about the 3D positions of the fiducial set 340defined within coordinate system 360 that was retrieved from the memoryof fixture 300, as the patterns are independent of the coordinate systemin which they are defined, provided that the pattern has no symmetry.The result of this matching will be the coordinate transform matrixrequired to rotate one coordinate system with respect to the othercoordinate system so as to match the pattern in coordinate system 160 tothe pattern in coordinate system 360.

In certain embodiments, the processor may determine that it is necessaryto identify a specific fiducial when multiple fiducials are in the firststate. When this occurs, the processor will cause the fiducial tomodulate its light in a fashion that can be detected by the sensor. Forexample, when the sensor is a CCD imager, the fiducial may turn on andoff at a rate that is slower than the frame rate of the imager. Inanother example, where the sensor is a PSD, the processor maysynchronize the “on” and “off” states of the fiducial with the reset ofthe PSD accumulator to maximize difference between “on” and “off” signalstrength. Alternately, the fiducial may adjust the intensity of theemitted light between two levels that can be distinguished by the pixeldetectors of the CCD sensor. Alternately, the fiducial may adjust thewavelength of the emitted light between two wavelengths that can bedistinguished by the pixel detectors of the CCD sensor.

The same methodology used to determine the position and orientation ofthe coordinate system 360 of the fixture 300 within the coordinatesystem 160 of the camera module 100 can be then used to determine theposition and orientation of the coordinate system 260 of the robotmodule 200 within the coordinate system 160. The processor of theelectronics module 400 retrieves information about the 3D positions ofthe fiducial set 240 in coordinate system 260 from the memory of therobot module 200. The processor manipulates the fiducials of fiducialset 240 in order to determine the location of each fiducial in thecoordinate system 160. The processor then uses the information about the3D positions of the fiducial set 240 in coordinate system 260 todetermine the coordinate transform matrix relating coordinate system 360to coordinate system 160.

Once the coordinate transformation matrices relating each of thecoordinate systems 260 and 360 to coordinate system 160 are determined,it is straightforward to create a coordinate transform matrix that mapspositions in the coordinate system 360 of the fixture 300 into thecoordinate system 260 of the robot module 200. Any point of interactionwith the fixture, for example the location of a receptacle configured toreceive a part to be processed, that is included in the informationcontained in the memory of fixture 300 can be retrieved and convertedinto coordinate system 260. This data can then be used by the processorthat controls the robot arm 220 to place an object in that location.

While the exemplary system 10 described herein associates certainmethods and activities with processors of specific modules, theinterconnection of the modules enables any function or algorithm to beexecuted by any processor in the system 10. For example, a portion ofthe processing of the output of the sensors of optical systems 120, 122may be performed by a processor within the optical systems 120, 122themselves, or by a separate processor contained in the enclosure 110 ofthe camera module 100, or by a processor in the electronics module 400.

The autonomous determination of relative positions of modules greatlysimplifies the set-up of a new configuration of workcell 10. An operatorsimply places the robot module 200 and one or more fixtures 300 withinthe FOV of camera module 100 and initiates the position determinationprocess. Information about the 3D positions of the fiducial set of eachmodule, defined within the coordinate system of the respective module,is retrieved from the memories of each module. The fiducials aremanipulated into various combinations of first and second states, e.g.“on” and “off,” and the outputs of the sensors are used to determine the3D positions of the fiducials in the coordinate system 160. These 3Dpositions are then used in conjunction with the information about the 3Dpositions of the fiducials within the various coordinate systems of themodules 300 to map specific locations on the fixtures into thecoordinate system of the robot module 200. When this process iscomplete, the workcell 10 notifies the operator that it is ready foroperation.

In certain embodiments, one or more of the modules 200, 300 comprises anaccelerometer configured to detect translational or rotational movementof the respective module. For example, motion of a module may be inducedby vibration of the work surface or the module during operation, contactbetween the robot arm 122 and the fixture 300, or an operator bumpinginto one of the modules. If a module moves, the workcell 10 will stopprocessing and repeat the position determination process to update thelocations on the fixtures within the coordinate system of the robotmodule 200. In most circumstances, the workcell does not need tophysically reset or return to a home position during the positiondetermination process. The robot arm 220 simply stops moving while theposition determination process is executed and then resumes operationupon completion.

In certain embodiments, workcell 10 is positioned on a flat work surface(not visible in FIG. 1) that constrains the possible orientations of thevarious modules relative to the camera module 100. In certainembodiments, it is sufficient to have only two fiducials on a modulevisible by the optical systems 120, 122.

In certain embodiments, there are 3 or more fiducials in the fiducialset of a module and the processor that is manipulating the fiducials andaccepting the information about the positions of the fiducial imagesfrom the optical systems 120, 122 is configured to utilize onlyfiducials that create images on the sensors, i.e. not use fiducials thatare hidden from the optical systems 120, 122. As long as a minimumnumber of the fiducials in the fiducial set are visible to the opticalsystems 120, 122, the position determination process will be successful.If it is not possible to obtain 3D position information from enoughfiducials to determine the position of a module within coordinate system160, the workcell 10 will notify the operator. In certain embodiments,the modules are configured to detect whether a fiducial is drawingcurrent or emitting light and when a fiducial is not able to move to thefirst state, e.g. does not turn on, notify the operator of this failure.

In certain embodiments, the camera module comprises multiple intensitydetectors that each have an output related to the total energy receivedover the FOV of that detector. Modulation of a fiducial between “on” and“off” states, and determination of the difference in the total receivedenergy when the fiducial is on and off provides a measure of the energyreceived from that fiducial. The processor of the electronics module 400calculates a distance from the camera module 100 to the fiducial basedon the ratio of the received energy to the emitted energy and solidgeometry. This creates a spherical surface centered at the entranceaperture of the intensity detector. In certain embodiments, the energyemitted by the fiducial and the geometry of the emitted light is part ofthe information contained in the memory of the module of the fiducialand downloaded with the 3D position information. With three intensitydetectors each providing a sphere of possible locations of the fiducial,intersection of the three spheres specifies a point within 3D spacewhere the fiducial is located.

FIGS. 2A-2B depict plan views of the robot module 200 from the work cellof FIG. 1 according to certain aspects of the present disclosure. Inthis embodiment, fiducial 244 is offset from the origin of coordinatesystem 240 by distance D1 in the R1 direction and distance D2 in the R2direction. Fiducial 242 is offset from fiducial 244 by a distance D3along vector 250 at an angle A1 from axis R1. Fiducial 246 is offsetfrom fiducial 244 by a distance D3 along vector 252 at an angle (A1+A2)from axis R1. The relative position of the fiducials 242, 244, 246 tothe origin of coordinate system 260 may be defined in other ways, forexample a direct vector from the origin to the fiducial (not shown inFIG. 2A), and the length and angle of that vector. From thisinformation, combined with the positions of some of fiducial set 240 inthe coordinate system 160 of the camera module 100, it is possible todetermine the position of the origin of coordinate system 260 incoordinate system 160 as well as the directional vectors in coordinatesystem 160 that are equivalent to directions R1, R2, R3.

FIG. 2B depicts the configuration parameters associated with the robotarm 220 defined in the coordinate system 260. In the embodiment, therobot arm 220 has a first degree of freedom around axis 222, which isparallel to coordinate direction R3. The position of axis 222 is definedby distances D5 and D6 in the R2 and R1 directions, respectively. Theangular position of the robot arm 220 is defined an axis 224 that isaligned with the segments of the robot arm 220 and rotated to an angleA3 with respect to direction R1. The position of the gripper 226 withrespect to the axis 222 is defined by the distance D7 along axis 224.This information, in conjunction with the knowledge of the position ofthe origin of coordinate system 260 and the directional vectors that areequivalent to directions R1, R2, R3 of coordinate system 260 incoordinate system 160, enables the position and angle of the gripper 226to be determined in coordinate system 160. Conversely, any positionknown in coordinate system 160 can be computed with reference to axis222 and angle A3.

The benefit of transferring the position of a fixture 300, and points ofinteraction of fixture 300, into the coordinate system 260 of robotmodule 200 is to provide the processor 270 with information that enablesthe processor 270 to move the gripper 226 to the point of interaction offixture 300. Accomplishing this position determination of the point ofinteraction of fixture 300 in coordinate system 260 through the use of aan automatic system and process enables the system 10 to functionwithout the modules 200, 300 being fixed to a common structure, thusspeeding and simplifying the set-up of the system 10. Once the positionof the fixture 300 is known in the coordinate system 260 of the robotmodule, and the transfer of the additional information regarding thepoints of interaction of the fixture 300 to the processor 270 and thetransformation of the positions of the points of interaction into thecoordinate system 260 is completed, the system is ready for operationwithout a need for the operator to teach or calibrate the system. Thisis discussed in greater detail with regard to FIGS. 5A-5B.

FIG. 3 depicts a plan view of a portion of a workcell 600 according tocertain aspects of the present disclosure. The camera module 100, robotmodule 200, and fixture 300 are positioned on a flat horizontal surface610. Optical system 120 has a FOV 124 that is delimited by the dashedlines emanating from the curved lens of optical system 120. Opticalsystem 122 has a FOV 126 that is delimited by the dashed lines emanatingfrom the curved lens of optical system 122. The effective FOV 130 isindicated by the shaded region where the FOVs 124, 126 overlap. The workarea of the workcell 600 is further defined by the focal range 140 ofthe optical systems 120, 122, delimited by the minimum focal distance142 and maximum focal distance 144 from the camera module 100. Incertain embodiments, the distances 142, 144 are defined from the centerof coordinate system 160. In certain embodiments, the distance 144 mayeffectively be infinitely far from the camera module 100.

When an operator positions the modules 200, 300 within the FOV 130, oneof the fiducials may be outside the FOV. In FIG. 3, fiducial 341 offiducial set 340 on fixture 300 is outside the shaded area of FOV 130.In addition, fiducial 343 is not visible by optical system 122, beinghidden behind the post 321. As long as two fiducials, in this examplefiducials 345, 347, are visible to both optical systems 120, 122 thenthe position of fixture 300 can be determined.

FIG. 4A depicts an exemplary optical design according to certain aspectsof the present disclosure. In the example binocular system 500, a lens510 and a sensor 512 are arranged with their centers positioned on anoptical axis (OA) 514 and separated by a distance 516 between thesensitive surface of the sensor 512 and the center of the lens 510.Sensor 512 has a polarity indicated by the “+” and “−” signs as to thepolarity of the output when a light spot is formed on the upper or lowerportions (in this 2D view) of the sensor 512. The set of lens 510 andsensor 520 may be considered as an optical system. This optical systemmay contain other optical elements, for example additional lenses,reflectors, baffles, stops, and filters, without departing from theconcepts disclosed herein. The system 500 also has a second opticalsystem comprising lens 540 and sensor 542 arranged on OA 544 andseparated by distance 546. Note that the sensors 512 and 542 have theirpolarities aligned in the same direction. The system 500 has a center502 with reference axes Y+, Y−, X+, and X−. In certain embodiments, thiscenter and reference axes correspond to the coordinate system 160 ofcamera module 100. A system OA 504 passes through the center 502, withthe OA 514 separated from the system axis 504 by distance 518 and the OA544 separated from the system axis 504 by distance 548. All OAs 504,514, are 544 are parallel and coplanar. When referring to positions,angles, centers, and other aspects of the sensors 512, 542 and similar,it is understood that this is equivalent to the respective aspects ofthe sensitive surface of the sensor. For example, a center of the sensoris equivalent to the center of the sensitive surface of that samesensor.

A target item 550, for example a fiducial, is positioned within the workarea of system 500. Exemplary rays 556 and 558 of light emanate from thetarget item 550 and respectively pass through the centers of lenses 510,540 until the ray 556 strikes the sensitive surface of sensor 512 at adistance 560 in the negative direction from the OA 514. The lens 510 andsensor 512 are arranged as a focusing system with a focal range thatencompasses the position of target item 550, and therefore other raysemanating from the same point on target item 550 that pass through lens510 will be refracted onto the same point of impact on the sensor 512.Ray 558 strikes the sensitive surface of sensor 542 at a distance 562 inthe positive direction from the OA 544.

In system 500, the absolute values of distances 560, 562 are differentby an amount that is proportional to the offset distance 552 of thetarget item 550 from the system axis 504 but both are still near or atthe limits of the sensitive region of sensors 512, 542. As a result,distance 554 is the minimum distance at which a target item is visiblein both optical systems and then only when on the system OA 504. Movingthe target item further away from the axis 504 will reduce the distance560 but increase the distance 562 and ray 558 will quickly move offsensor 542 as distance 552 increases. This results in significantportions of the sensors 512, 542 being un-useable and a consequentreduction in the work area of system 500.

FIG. 4B depicts another exemplary optical design according to certainaspects of the present disclosure. Binocular system 501 is amodification of system 500, wherein the OAs 514, 544 are each rotatedtoward the system axis 504 and are not parallel to each other. Incertain embodiments, the included angle 506 between the two OAs 514, 544is greater than zero and less than 180 degrees. In certain embodiments,the included angle 506 between the two OAs 514, 544 is in the range of5-15 degrees. This is contrary to the design rules for optical systemsdesigned for use by humans as the human eye and brain are wired toproperly form merged images only when the OAs 514, 544 are parallel. Inthis example, distances 561, 563 are equal as the target item 550 is nowon the system axis 504 and are approximately equal to distance 562 ofFIG. 4A, which was the larger of the distances 560, 562. In system 501,the distance 565 from the center 502 to the target item 550 is shorterthan distance 554 of system 500, thereby bringing the work area closetto the camera module 100. In certain embodiments, the two OAs 514, 544pass through the center of the sensitive areas of the respective sensors512, 542, the sensitive surfaces of which are perpendicular to the OAs514, 544. As a result, the sensitive surface of sensor 542 is notparallel to the sensitive surface of sensor 512.

The angled OAs 514, 544 increase the total size of the work and make useof a larger portion of the sensitive surfaces of sensors 512, 542. Incertain embodiments, the amounts that the axes 514, 544 are rotatedtoward the system axis 504 are not equal.

FIG. 4C depicts a third exemplary optical design according to certainaspects of the present disclosure. In system 600, the sensors 618, 628are offset outward by distances 618, 628, respectively, from the OAs611, 621 that pass through the centers of lenses 610, 620. In certainembodiments, the sensitive surfaces of sensors 612, 622 are parallel. Incertain embodiments, the sensitive surfaces of sensors 612, 622 arecoplanar. The lines 614, 616 define the paths of rays that will strikethe outer edges of sensor 612 and therefore define FOV 630. Similarly,lines 624, 626 define the paths of rays that will strike the outer edgesof sensor 622 and therefore define FOV 632. The area of overlap of FOVs630, 632 define the working area 634 of system 600. The area 634 iswider in the front, i.e. the edge towards the sensors 618, 628, andcloser to the sensors 618, 628 than the equivalent area of system 500,i.e. without the offset of sensors 618, 628. The shape of work area 634is much more useful than the triangular work area 130 of FIG. 3.Although there is a loss of coverage of the outer, rear corners of workarea 634, this is generally not useful space in workcells.

FIGS. 5A-5B depict an exemplary fixture 700 and its associated spatialenvelope 750 according to certain aspects of the present disclosure.Fixture 700 is a simplistic device, where a ball 701 is dropped into afunnel 712, passes into the body 710 of the fixture 700 and is processedin some fashion, then exits through aperture 714 into a trough 720 andstops in position 701A. A coordinate system 730 with three orthogonalaxes X, Y, Z is defined for the fixture 700. In certain embodiments,coordinate system 730 is also used to define the 3D positions offiducials (not shown in FIG. 5A). If this fixture 700 were part of arobotic workcell, it would be desirable to have the robot drop the balls701 into the funnel 712 and then remove the processed balls fromlocation 701A.

FIG. 5B depicts a spatial envelop 750 that is defined to genericize theinterface of fixtures to a robot module, such as the robot module 200 ofFIG. 1. First, a “keep-out” zone is defined in coordinate system 730that encompasses the fixture 700. This serves dual purposes, in thatacceptance of a fixture may include a comparison to this envelope 752,which would be provided to the fixture supplier in the procurementspecifications, to ensure that the fixture is as designed. Secondly, theprocessor that is controlling a robot arm, for example the robot arm 220of FIG. 1, can compare the position of the end effector or other pointson the arm to the volume of the keep-out zone and adjust the robot armposition and path to avoid entering the keep-out zone. The coordinatesof the corners of the keep-out zone are defined in the coordinate system730 and are included in the information stored in the memory of fixture700. The keep-out zone need not be a rectilinear solid and may have anyform and include curves, protrusions, and cavities. These coordinatesand characteristics of the keep-out zone may be downloaded, for exampleby the processor of electronics module 400 of FIG. 1, and passed to theprocessor controlling the robot arm 220.

The example envelope 750 also defines interaction points, e.g. an “inputlocation” and an “output location” on the surface of the keep-out zones,as guidance for use of the fixture 700, for example by robot module 200.In this example, the input location is defined in two exemplary ways.First, a circular opening 754 is defined on the surface of the keep-outzone where the robot arm is to place an unprocessed ball 701. The planarinput location 754 is defined by location references 762, 764 to thecoordinate system 730 and diameter 758. In some embodiments, the inputlocation is defined as a volume 756, wherein the robot arm is to placethe ball 701 at any point within this volume 756 and then release theball 701. The volume 756 is defined by location references 762, 764 tothe coordinate system 730 as well as the height 760 and diameter 758 ofthe volume 756.

The output location is also shown in two exemplary ways. First, acircular exit 770 is defined on the surface of the keep-out zone 752 andlocated by locations references 774, 776 and diameter 778. An exitvector 772 is also defined to indicate the direction of motion of theball 701 when it passes through the exit 770. A second example ofdefining an output location is the final location 780, which indicatesthe position 701A from FIG. 5A where the ball 701 will come to a stop.In some embodiments, an approach vector 782 is defined to indicate fromwhat direction the end effector should approach the location 780. Theinformation about the characteristics of the input and output locationsis also included in the information stored in the memory of fixture 700and may be downloaded with the other information.

Other types of interaction points may be defined as appropriate for thepurpose of the workcell and types of parts to be handled in the workcell. For example, multiple holding locations may be defined on a tray,each location designated as an interaction point. Interaction points maydepend on the status or content of a module, for example a tray may havea first set of interaction points for a first layer of parts that restdirectly on the tray and a second set of interaction points for a secondlayer of parts that is stacked upon the first layer of parts and areonly usable when the first layer of parts is in place. A tool may haveloading and unloading interaction points, or a single interaction pointthat is both a loading and unloading location.

Defining the keep-out zone and input and output locations and vectorsenables this information to be downloaded from the fixture 700 to aprocessor in the workcell that controls the robot arm so as to automatethe programming of the robot arm to interact with fixture 700. Theinformation contained in the memory of fixture 700 comprises the 3Dlocation of the fiducials, the characteristics of the keep-out zone, andthe characteristics of the input and output locations and approachvectors as well as other information such as a fixture model, serialnumber, dates of manufacture or latest service, or any otherfixture-related data. In certain embodiments, this information is storedin a defined data structure that is common to all fixtures. In certainembodiments, the information is provided as data pairs, the first dataitem being a data identifier for a particular parameter and the seconddata item being the value or values of that parameter. In certainembodiments, data may be stored according to any style of storage thatenables retrieval and identification of the data elements in the memory.

One of the advantages of providing information about the fixture in thisgeneric way is that generic commands are sufficient to instruct therobot on how to interact with the fixture. For example, a command “placeitem in input location” is enough, when combined with the informationfrom the fixture that defines the location of the fixture relative tothe robot arm, the position of the input location, and the vector to beuse to approach the input location, to cause the robot to place the item(which it presumably picked up in a previous step) in the desired inputlocation of the fixture. This simplifies the programming and avoidshaving to program the workcell as to the precise location of the inputlocation for this set-up.

In some embodiments, the characteristics of a single type of fixture arestored as data set in a library that is available, for example, to theprocessor of electronics module 400. This library may be resident in thememory of electronics module 400, in the memory of robot module 200, ona remote memory accessible by the processor of electronics module 400over a network, or in any other storage location or medium whereinformation can be stored and retrieved. Each data set will have aunique identifier that is also stored in the memories of modules of thattype of fixture. The processor need only retrieve the identifier from afixture, and retrieve the data set associated with that type of fixturefrom the library. Retrieval of the identifier may be by scanning amachine-readable code such as a barcode, observing a machine-readablecode by one of the optical systems of camera module 100 and parsing thatportion of the image, scanning a Radio Frequency Identification (RFID)device, or through interaction with a processor or other electronicdevice over a wired or wireless communication link. In some embodiments,the library is available on a central server, for example a cloud-baseddata server, to which the processor can be communicatively coupled. Incertain embodiments, the library is downloaded to a memory of theelectronics module 400 and periodically updated as new fixtures arereleased. In certain embodiments, the library is stored on the computer500 of FIG. 1.

A module 700 may have only output locations. For example, a tray ofparts may be provided to the work cell. In some embodiments, the traymay have a barcode label printed in a location visible to the cameramodule 100. The memory of the electronics module 400 may includeinformation about the size, shape, fiducial locations, and interactionpoints of the tray. After the position of the tray is determined by thesystem 10 and the information regarding access points and locations istransferred to the processor 270, the processor 270 can then direct therobot arm 220 to remove parts from the tray. When the tray is empty, aswill be known by the processor 270 after it has interacted with each ofthe interaction points of the tray, e.g. removed a part from each of theinteraction points, the workcell 10 can signal for a new tray of partsto be provided, for example by sending a signal to another system toremove the empty tray and provide a new tray with parts.

This application includes description that is provided to enable aperson of ordinary skill in the art to practice the various aspectsdescribed herein. While the foregoing has described what are consideredto be the best mode and/or other examples, it is understood that variousmodifications to these aspects will be readily apparent to those skilledin the art, and the generic principles defined herein may be applied toother aspects. It is understood that the specific order or hierarchy ofsteps or blocks in the processes disclosed is an illustration ofexemplary approaches. Based upon design preferences, it is understoodthat the specific order or hierarchy of steps or blocks in the processesmay be rearranged. The accompanying method claims present elements ofthe various steps in a sample order, and are not meant to be limited tothe specific order or hierarchy presented. Thus, the claims are notintended to be limited to the aspects shown herein, but is to beaccorded the full scope consistent with the language claims.

Headings and subheadings, if any, are used for convenience only and donot limit the invention.

Reference to an element in the singular is not intended to mean “one andonly one” unless specifically so stated, but rather “one or more.” Useof the articles “a” and “an” is to be interpreted as equivalent to thephrase “at least one.” Unless specifically stated otherwise, the terms“a set” and “some” refer to one or more.

Terms such as “top,” “bottom,” “upper,” “lower,” “left,” “right,”“front,” “rear” and the like as used in this disclosure should beunderstood as referring to an arbitrary frame of reference, rather thanto the ordinary gravitational frame of reference. Thus, a top surface, abottom surface, a front surface, and a rear surface may extend upwardly,downwardly, diagonally, or horizontally in a gravitational frame ofreference without limiting their orientation in other frames ofreference.

Although the relationships among various components are described hereinand/or are illustrated as being orthogonal or perpendicular, thosecomponents can be arranged in other configurations in some embodiments.For example, the angles formed between the referenced components can begreater or less than 90 degrees in some embodiments.

Although various components are illustrated as being flat and/orstraight, those components can have other configurations, such as curvedor tapered for example, in some embodiments.

Pronouns in the masculine (e.g., his) include the feminine and neutergender (e.g., her and its) and vice versa. All structural and functionalequivalents to the elements of the various aspects described throughoutthis disclosure that are known or later come to be known to those ofordinary skill in the art are expressly incorporated herein by referenceand are intended to be encompassed by the claims. Moreover, nothingdisclosed herein is intended to be dedicated to the public regardless ofwhether such disclosure is explicitly recited in the claims. No claimelement is to be construed under the provisions of 35 U.S.C. § 112,sixth paragraph, unless the element is expressly recited using thephrase “means for” or, in the case of a method claim, the element isrecited using the phrase “operation for.”

A phrase such as an “aspect” does not imply that such aspect isessential to the subject technology or that such aspect applies to allconfigurations of the subject technology. A disclosure relating to anaspect may apply to all configurations, or one or more configurations. Aphrase such as an aspect may refer to one or more aspects and viceversa. A phrase such as an “embodiment” does not imply that suchembodiment is essential to the subject technology or that suchembodiment applies to all configurations of the subject technology. Adisclosure relating to an embodiment may apply to all embodiments, orone or more embodiments. A phrase such as an embodiment may refer to oneor more embodiments and vice versa.

The word “exemplary” is used herein to mean “serving as an example orillustration.” Any aspect or design described herein as “exemplary” isnot necessarily to be construed as preferred or advantageous over otheraspects or designs.

All structural and functional equivalents to the elements of the variousaspects described throughout this disclosure that are known or latercome to be known to those of ordinary skill in the art are expresslyincorporated herein by reference and are intended to be encompassed bythe claims. Moreover, nothing disclosed herein is intended to bededicated to the public regardless of whether such disclosure isexplicitly recited in the claims. No claim element is to be construedunder the provisions of 35 U.S.C. § 112, sixth paragraph, unless theelement is expressly recited using the phrase “means for” or, in thecase of a method claim, the element is recited using the phrase “stepfor.” Furthermore, to the extent that the term “include,” “have,” or thelike is used in the description or the claims, such term is intended tobe inclusive in a manner similar to the term “comprise” as “comprise” isinterpreted when employed as a transitional word in a claim.

Although embodiments of the present disclosure have been described andillustrated in detail, it is to be clearly understood that the same isby way of illustration and example only and is not to be taken by way oflimitation, the scope of the present invention being limited only by theterms of the appended claims.

What is claimed is:
 1. A fixture comprising: a body, and a plurality offiducials fixedly coupled to the body, wherein at least one of theplurality of fiducials comprises a first state and a second state and isconfigured to selectively move to one of the first state and the secondstate, wherein the fiducials are arranged such that knowledge of the 3Dpositions of the plurality of fiducials in an observing coordinatesystem is sufficient to determine a first position and a firstorientation of the body in the observing coordinate system.
 2. Thefixture of claim 1, wherein each of the plurality of fiducials comprisesa first state and a second state and is configured to selectively moveto one of the first state and the second state.
 3. The fixture of claim2, wherein the fixture is configured to selectively cause a portion ofthe plurality of fiducials to move to the first state and cause a restof the plurality of fiducials to move to the second state.
 4. Thefixture of claim 3, wherein the fixture is configured to receive asignal and, upon receipt of the signal, cause the portion of theplurality of fiducials to move to the first state and cause the rest ofthe plurality of fiducials to move to the second state.
 5. The fixtureof claim 1, wherein the plurality of fiducials are disposed so as todefine a local coordinate system.
 6. The fixture of claim 5, wherein theplurality of fiducials are disposed so as to enable determination of asecond position and a second orientation of the local coordinate systemin the observing coordinate system by a binocular camera module having adefined arrangement in the observing coordinate system.
 7. The fixtureof claim 1, wherein the first state comprises emitting a first amount oflight and the second state comprises emitting a second amount of lightthat is different from the first amount of light.
 8. The fixture ofclaim 7, wherein the at least one of the plurality of fiducials isconfigured to selectively modulate the amount of light emitted while inthe first state.
 9. The fixture of claim 1, wherein the first statecomprises emitting light at a first frequency.
 10. The fixture of claim9, wherein the first frequency is separated from the range of 50-60 Hzsufficient to differentiate the light emitted by the fiducial from lightemitted by a common light source.